Predicting Part-of-Speech Information about Unknown Words using Statistical Methods
نویسنده
چکیده
This paper examines the feasibility of using sta tistical methods to train a part of speech pre dictor for unknown words By using statistical methods without incorporating hand crafted linguistic information the predictor could be used with any language for which there is a large tagged training corpus Encouraging re sults have been obtained by testing the predic tor on unknown words from the Brown corpus The relative value of information sources such as a xes and context is discussed This part of speech predictor will be used in a part of speech tagger to handle out of lexicon words
منابع مشابه
Tagging Unknown Words using Statistical Methods
This paper examines the feasibility of using statistical methods to train a part-of-speech tagger, particularly with respect to unknown words. Training a part-of-speech tagger on a tagged corpus, without incorporating hand-crafted linguistic information, allows that tagger to be used for any language. The use of statistical methods has given encouraging results in experiments performed using th...
متن کاملHybrid Models for Chinese Unknown Word Resolution Dissertation
Word segmentation, part-of-speech (POS) tagging, and sense tagging are important steps in various Chinese natural language processing (CNLP) systems. Unknown words, i.e., words that are not in the dictionary or training data used in a CNLP system, constitute a major challenge for each of these steps. This dissertation is concerned with developing hybrid models that effectively combine statistic...
متن کاملGuessing Parts-of-Speech of Unknown Words Using Global Information
In this paper, we present a method for guessing POS tags of unknown words using local and global information. Although many existing methods use only local information (i.e. limited window size or intra-sentential features), global information (extra-sentential features) provides valuable clues for predicting POS tags of unknown words. We propose a probabilistic model for POS guessing of unknow...
متن کاملMultilingual Word Segmentation and Part - of - Speech Tagging : a Machine Learning Approach Incorporating Diverse Features ∗
The aim of this dissertation is to study statistical methods for multilingual word segmentation and POS tagging with high accuracy. Word segmentation and part-of-speech (POS) tagging are fundamental language analysis tasks in natural language processing, and used in many applications. Existence of unknown words is a large problem in these tasks and they need to be properly handled. We attempt t...
متن کاملPhrase Break Prediction Using a Finite State Transducer
This paper presents a method for phrase break prediction using a finite state transducer. In the literature, several algorithms have been proposed using statistical techniques for predicting phrase breaks. Some of these methods rely on linguistic information, such as syllables, words, part-of-speech, accents, etc. Our proposal is a probabilistic finite state transducer to convert part-ofspeech ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 1998